Motion training device, program, and display method

ABSTRACT

A motion training device includes: a display on which a user can visually recognize a training target site as a part of a body part of the user in the state where motion images is displayed; and a controller which causes the display to display motion images that appears as if the training target site is moving, at a different position from a position of the training target site of the user.

BACKGROUND Technical Field

The present disclosure relates to motion training.

Related Art

JP-A-2004-298430 discloses a technique of allowing a patient to visuallyrecognize motion images which reproduces a missing body part asrealistically as possible for the treatment of phantom limb pain.JP-A-2004-298430 discloses applying this technique to the treatment ofparalysis.

In the related-art technique, the user is allowed to visually recognizemotion images that are as realistic as possible. Therefore, when thetechnique is used for motion training of a body part that is notmissing, the motion images is displayed so as to be superimposed on atraining target site that is a body part of the user. Thus, it isdifficult to compare the body part of the user and the motion images,and therefore the training is not effective.

SUMMARY

An advantage of some aspect of the disclosure is to solve the problemsdescribed above, and the disclosure can be implemented as the followingconfigurations.

An aspect of the disclosure is directed to a motion training deviceincluding: a display on which a user is capable of visually recognizinga training target site as a part of a body part of the user in the statewhere motion images is displayed; and a controller which causes thedisplay to display motion images that appears as if the training targetsite is moving, at a different position from a position of the trainingtarget site of the user. With this configuration, the motion images aredisplayed at a different position from the position of the trainingtarget site of the user. Therefore, it is easier to compare the bodypart of the user with the motion images.

The motion training device may be a head-mounted display device. Withthis configuration, the device is conveniently portable and enablestraining without needing a large space.

In the motion training device, the display may be an opticaltransmission-type display. With this configuration, the user canactually see the training target site as a body part of the user.Therefore, it is easier to compare the body part of the user with themotion images.

The motion training device may further include a camera which capturesan image of a real space. The display may be a video see-throughdisplay. The controller may cause the display to display the trainingtarget site whose image is picked up by the camera, corresponding to aposition of the training target site in the real space. With thisconfiguration, the display is a video see-through display and blocksunwanted external light. Therefore, the user can concentrate ontraining.

In the motion training device, the training target site may be one hand.With this configuration, the user can carry out training of the one handeffectively.

In the motion training device, the training target site may be bothhands. The controller may cause the motion images to be displayedbetween both hands of the user. With this configuration, in the casewhere the target site is both hands, the motion images can be displayedat a position which the user can easily see, considering the positionsof both hands of the user.

In the motion training device, the controller may cause the trainingtarget site displayed as the motion images to be displayed so as to looksmaller than the training target site of the user. With thisconfiguration, the motion images are easier to see.

In the motion training device, the controller may cause the motionimages to be displayed so as to avoid overlapping the training targetsite of the user. With this configuration, it is easier to compare thebody part of the user with the motion images.

The disclosure can be implemented in various configurations other thanthose described herein. For example, the disclosure can be realized inconfigurations such as a display method, a program for realizing thismethod, and a non-temporary storage medium storing this program.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure will be described with reference to the accompanyingdrawings, wherein like numbers reference like elements.

FIG. 1 shows the configuration of a Head Mounted Display (HMD).

FIG. 2 shows the configuration of a left-eye display.

FIG. 3 is a functional block diagram of the HMD.

FIG. 4 shows the state of a user wearing the HMD.

FIG. 5 is a flowchart showing display processing.

FIG. 6 shows the state where both hands are open during the displayprocessing.

FIG. 7 shows the state where both hands are closed during the displayprocessing.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

FIG. 1 shows the configuration of a Head Mounted Display (HMD) 10. Theterm “HMD” is an abbreviation of head mounted display. The HMD 10executes display processing, described later, and thus realizes adisplay method for motion training and functions as a motion trainingdevice. Motion training is carried out, for example, for rehabilitationof a hand (hereinafter also referred to as rehab).

The HMD 10 is an optical transmission-type (optical see-through) displayon which the user can simultaneously see both a virtual image and realspace.

The HMD 10 includes a display device 20 shaped like eyeglasses, and acontroller 70. The display device 20 and the controller 70 are connectedvia a wire or wirelessly so as to be able to communicate with eachother. In this embodiment, the display device 20 and the controller 70are connected via a cable 90. The controller 70 communicates a signal ofan image (image signal) and a signal for control (control signal) withthe display device 20 via the cable 90.

The display device 20 has a left-eye display 30L and a right-eye display30R. The left-eye display 30L includes a left-eye output 32L, a left-eyelight guide 34L (FIG. 2), a left-eye reflector 36L, and a left-eye shade38L. The right-eye display 30R includes a right-eye output 32R, aright-eye light guide (not shown), a right-eye reflector 36R, and aright-eye shade 38R. The right-eye light guide has a configurationsimilar to that of the left-eye light guide 34L.

FIG. 2 shows details of the configuration of the left-eye display 30L.FIG. 2 shows the left-eye display 30L, as viewed from directly above.The left-eye output 32L is arranged at the base of the temple of theeyeglasses and includes a left-eye image generator 321L and a left-eyeprojection system 322L.

The left-eye image generator 321L includes a left-eye backlight lightsource BL and a left-eye light modulation element LM. In thisembodiment, the left-eye backlight light source BL is made up of a setof light sources corresponding to the individual light emission colorsof red, green, and blue. For example, a light emitting diode (LED) orthe like can be used as each light source. In the embodiment, theleft-eye light modulation element LM is made up of a liquid crystaldisplay device, which is a display element.

The left-eye display 30L functions as follows. When an image signal forthe left eye is inputted to the left-eye image generator 321L from thecontroller 70 (see FIG. 1), the individual light sources of the left-eyebacklight light source BL emit red light, green light, and blue light.The red light, green light, and the blue light emitted from theindividual light sources are dispersed and become incident on theleft-eye light modulation element LM. The left-eye light modulationelement LM spatially modulates the red light, the green light, and theblue light which are projected thereon, according to an image signalinputted to the left-eye image generator 321L from the controller 70,and thus emit an image light corresponding to the image signal.

The left-eye projection system 322L is made up of, for example, aprojection lens set, and turns the image light emitted from the left-eyelight modulation element LM of the left-eye image generator 321L into aparallel luminous flux. The image light, turned into the parallelluminous flux by the left-eye projection system 322L, becomes incidenton the left-eye light guide 34L.

The left-eye light guide 34L guides the image light from the left-eyeprojection system 322L, to a semitransparent reflection surface of atriangular prism provided in the left-eye reflector 36L. Of the frontand back sides of the semitransparent reflection surface, the sidefacing the left eye EY of the user when the user wears the HMD isprovided with a reflection coating such as a mirror layer. This surfacewith the reflection coating totally reflects the image light guided tothe semitransparent reflection surface at an image extraction area ofthe left-eye reflector 36L. The reflected image light enters the lefteye EY of the user. Consequently, the left eye EY of the user sees avirtual image. In this embodiment, making the user perceive a virtualimage is also expressed as displaying an image.

At least part of the light incident on the left-eye reflector 36L fromthe real space (e.g., the surrounding environment) is transmittedthrough the semitransparent reflection surface of the left-eye reflector36L and is guided to the left eye EY of the user. Thus, the user seesthe image displayed by the left-eye output 32L and the optical imagefrom the real space simultaneously so that the image displayed by theleft-eye output 32L appears superimposed over the image from the realspace.

As shown in FIG. 1, the right-eye display 30R has a configurationsimilar to that of the left-eye display 30L with left-right symmetry,and functions similarly to the left-eye display 30L.

In accordance with the foregoing configuration, the user cansimultaneously see the image displayed in the image extraction area ofthe display device 20 and the real space. The image thus displayedprovides the user with augmented reality (AR).

The left-eye shade 38L is arranged on the side of the left-eye lightguide 34L that is opposite to the left eye EY of the user. The left-eyeshade 38L in the embodiment is removable. The left-eye shade 38L blockslight so that the light will not be incident on the semitransparentreflection surface from the real space. Therefore, when the left-eyeshade 38L is attached, the user can clearly see the image displayed bythe left-eye output 32L.

In the display device 20, a camera 51 is provided at a positioncorresponding to the glabella of the user when the user wears thedisplay device 20 (proximate the bridge/top bar area of the HMD 10).Therefore, in the state where the user wears the display device 20 onthe head, the camera 51 picks up an image of the real space in thedirection in which the user faces. The camera 51 in this embodiment is asingle-lens camera. The camera 51 in another configuration can be astereo camera.

The controller 70 controls the display device 20. The controller 70includes a touch pad 72 and an operation buttons 74. The touch pad 72detects a touch operation on an operation surface of the touch pad 72and outputs a signal corresponding to the detection content. Varioustouch pads such as electrostatic types, pressure detection types, andoptical types can be employed as the touch pad 72. The operation buttons74 include various operation buttons, and detect inputs on therespective operation buttons and output signals corresponding to thedetection contents. The controller 70 is merely exemplary and othertypes of controls and input mechanisms may be substituted therefore.

FIG. 3 is a functional block diagram of the HMD 10. The controller 70includes a CPU 80, storage 82, a motion model database 84, an inputinformation acquirer 86, and a power supply 88. The individualcomponents are connected to each other via a bus or the like.

The storage 82 is made up of a ROM, RAM, DRAM, hard disk or the like.Various programs including an operating system (OS) and a program fordisplay processing are stored in the storage 82.

The motion model database 84 is a database in which motion models areaccumulated. A motion model is motion images data which models a motionof a training target site. In this embodiment, motion models for theleft hand and motion models for the right hand are accumulated inadvance.

The input information acquirer 86 includes the touch pad 72 and theoperation buttons 74. The input information acquirer 86 receives inputsignals corresponding to the detection contents from the touch pad 72and the operation buttons 74. The power supply 88 supplies electricityto the components provided in the controller 70 and the display device20.

The CPU 80 reads out and executes the programs stored in the storage 82and thereby realizes various functions. Specifically, the CPU 80executes processing corresponding to a detection content of an operationfrom the input information acquirer 86 when the detection content isinput, reads data from and writes data to the storage 82, and controlsthe supply of electricity to each component from the power supply 88.(In other words, the user operates the input information acquirer 86,the input information is detected, the detection content is input to theCPU 80, and the CPU 80 executes processing corresponding to thedetection content.

The CPU 80 reads out and executes programs for display processing storedin the storage 82 and thereby executes the display processing. Theexecution of the display processing realizes the display method formotion training. The display processing is configured to support themotion training of the user by displaying motion images of a body partwith a functional disorder. The user carries out motion training for therehabilitation of a paralyzed body part based on the displayed motionimages.

FIG. 4 shows the state where the user is wearing the HMD 10. The HMD 10in this embodiment is employed on the assumption that the user has afunctional disorder on at least one hand. The “hand” in this embodimentrefers to the entirety of the site from the wrist to the distal end ofthe hand.

The movement to be a target of motion training in the embodiment is theflexion and extension of the IP joint or the MP joint. The IP joint isthe first joint in the case of the first finger (thumb) (the distalinterphalangeal joint), and the first and second joints in the case ofthe second to fifth fingers (the distal interphalangeal joint and theproximal interphalangeal joint). The MP joint is the joint situated atthe boundary between the finger and the palm (metacarpophalangealjoint).

The finger to be a rehabilitation target is at least one of the first tofifth fingers of the hand including the finger with a functionaldisorder. Hereinafter, the hand including the finger with a functionaldisorder is called the “disabled hand FH”, and the hand without afunctional disorder is called the “normal hand NH”. The term “without afunctional disorder” means that the hand is not a rehabilitation target,and in practice, this hand may have a slight functional disorder. In thecase of the user shown in FIG. 4, the right hand is a normal hand NH andthe left hand is a disabled hand FH.

FIG. 5 is a flowchart showing the display processing described above.The display processing is started in response to a start instructioninput via the operation buttons 74. The display processing is executedrepeatedly until an end instruction is inputted via the operationbuttons 74. The operation buttons 74 are operated by the user or atraining assistant.

First, the CPU 80 detects the positions of both hands of the user(S110). Specifically, the left and right hand parts are extractedrespectively from an image picked up by the camera 51, and the contoursthereof are specified two-dimensionally.

Next, the CPU 80 determines whether motion images can be displayedbetween both of the hands or not (S120). A specific determination methodwill be described later. If motion images can be displayed between bothhands (YES in S120), motion images is displayed between both hands(S130) and the processing returns to S110.

FIGS. 6 and 7 show the states of images perceived by the user during thedisplay processing. FIG. 6 shows the state where both hands in an openstate are displayed as motion images IMG and where both hands of theuser are open, too. FIG. 7 shows the state where both hands in a closedstate are displayed as motion images IMG and where both hands of theuser are closed, too. The user visually recognizes the normal hand NHand the disabled hand FH through the semitransparent reflection surfaceof the display device 20. The motion images IMG in the embodiment areperceived by the user as three-dimensional images.

Since the motion images IMG are displayed between both hands of theuser, the motion images IMG are displayed at different positions thanthe positions of both hands of the user. In addition, since the motionimages IMG are displayed between both hands of the user, the motionimages IMG are displayed so as to avoid overlapping either hand of theuser. The term “avoid overlapping” means that there are no overlappingsites.

In FIGS. 6 and 7, the contour of a display area R1 is shown. The displayarea R1 is an area where an image can be displayed. However, the usercannot clearly see the contour of the display area R1. A rectangulararea R2 (described later) is a virtual area at which the real space isinvisible to the user.

A direction which is along a side of the rectangle as the contour of thedisplay area R1 and in which the right hand and the left hand of theuser are arranged next to each other is defined as a width direction. Adirection along a side of the rectangle as the contour of the displayarea R1 and orthogonal to the width direction is defined as a heightdirection. In the height direction, the fingertip side of the hands ofthe user is referred to as “top”, and the wrist side of the hands of theuser is referred to as “bottom”.

The positional relationship between the display area R1, and the normalhand NH and the disabled hand FH, is determined by the posture of theuser. The posture refers to the three-dimensional positionalrelationship between the head of the user, and the normal hand NH andthe disabled hand FH.

The motion images IMG are configured to show serial movements such asalternating between closing the hands and opening the hands. While thedetermination of YES is repeated in S120, the display of the motionimages IMG is continued.

The user is instructed in advance to close and open his/her hands alongwith the movements of the motion images. The disabled hand FH is both arehabilitation target site and a training target site. The normal handNH is not a rehabilitation target site but is a training target site.The motion training of the normal hand NH as well as the disabled handFH enhances the effect of the rehabilitation of the disabled hand FH.

The motion images IMG are displayed so that the height H2 of the motionimages IMG is shorter than the height H3 of each hand. The height H3 ofeach hand is a parameter indicating at what height the hand appears tothe user. Specifically, H3 is defined as a distance in the heightdirection and is equal to the distance from the topmost site of the handto the wrist. Consequently, the hands displayed as the motion images IMGare displayed so that they look smaller than the normal hand NH and thedisabled hand FH.

The height H3 of the hand can change according to the posture of theuser. In this embodiment, the height H2 of the motion images IMG in theembodiment is a fixed value. Therefore, the height H2 of the motionimages IMG is decided in advance so as to be shorter than the height H3at its minimum. The height H3 at its minimum is the height H3 of thehand in the state where the arm is fully extended. In anotherconfiguration, the height H2 may be a variable value and may be set bythe training assistant.

Next, the determination in S120 will be described. In the embodiment,the motion images IMG are displayed between both hands so as not tooverlap either hand, as described above. In S120, whether there is asufficient two-dimensional space for displaying the motion images IMG insuch a manner or not is determined. Specifically, if the rectangulararea R2 satisfies all of the following conditions, the CPU 80 determinesYES in S120.

A condition “a” is that the width W2 of the rectangular area R2 isentirely within an overlap width W1. The overlap width W1 refers to thewidth of an overlapping area among the space between both hands and thedisplay area R1.

A condition “b” is that each of the left and right hands is within therange of the display area R1 in the height direction.

The condition a is a condition for displaying the motion images IMGbetween the normal hand NH and the disabled hand FH so as not to overlapeither of the normal hand NH or the disabled hand FH.

The condition b is a condition for displaying the motion images IMG nearthe normal hand NH and the disabled hand FH and thus making it easier tocompare the motion images IMG with the normal hand NH and the disabledhand FH.

The positions where the motion images IMG are displayed are decided asfollows, using the conditions a and b. With respect to the widthdirection, the center of the rectangular area R2 is set on the center ofthe overlap width W1. With respect to the height direction, the centerof the rectangular area R2 is set on the center of a line from the topend to the bottom end of the left and right hands. The top end of theleft and right hands refers to the position of the higher one of the topend of the left hand and the top end of the right hand. The bottom endof the left and right hands refers to the position of the lower one ofthe left wrist and the right wrist.

The overlap width W1, the width W2 of the rectangular area R2, and theheight H3 of the hands change every moment. Therefore, the determinationin S120 is carried out based on the state of these parameters at eachtime point (e.g., continuously). If the detection of the positions ofboth hands fails, NO is determined in S120. For example, if one of thehands is located outside the image pickup range of the camera, theposition of that hand cannot be detected.

If it is determined that the motion images IMG cannot be displayedbetween both hands (NO in S120), the user is prompted to adjust thepositions of the head and both hands (S140). Specifically, a characterstring such as “Move both hands apart from each other and look betweenboth hands” is displayed as an image. Subsequently, the processingreturns to S110.

According to the embodiment described above, motion training targetingboth hands can be carried out effectively, using the see-through HMD 10.Also, since the HMD 10 is conveniently portable and does not take alarge space, motion training can be easily carried out at home or thelike.

The disclosure is not limited to the embodiment, examples andmodifications in this specification and can be realized with variousconfigurations without departing from the scope of the disclosure. Forexample, technical features in the embodiment, examples andmodifications corresponding to technical features in each configurationdescribed in the summary section can be suitably replaced or combined inorder to solve a part or all of the foregoing problems or in order toachieve apart or all of the foregoing advantageous effects. Suchtechnical features can be suitably deleted unless described as essentialin the specification. For example, the following examples can be given.

If both hands are disabled hands FH, the rehabilitation and trainingtarget sites may be both hands.

The training target site may be other than both hands. For example, thetraining target site may be one hand as a disabled hand. Alternatively,the training target site may be both legs or one leg, or may be one footor both feet. The leg in this case refers to a site from the thigh tothe distal end.

The motion images may be displayed at a different position from theposition of the training target site of the user. For example, themotion images may be displayed so that it overlaps with the trainingtarget site of the user. That is, apart of the motion images may bedisplayed so as to overlap the training target site of the user.Alternatively, if the training target site is both hands, the motionimages may be displayed at a position that is not between both hands(for example, above each of the left and right hands). The motion imagesmay be of parts other than both hands. For example, the motion imagesmay be of one hand as a disabled hand. Alternatively, the motion imagesmay be of both legs or one leg, or may be of one foot or both feet. Ifthe motion images are of one hand, the position of the one hand isdetected and whether the motion images can be displayed at a positionnot overlapping the one hand or not is determined. If the motion imagescan be displayed at a position not overlapping the one hand, the motionimages is displayed at the position not overlapping the one hand.

The motion training device may be a video see-through HMD. In the casewhere the video see-through HMD is used, a training target site as abody part of the user whose image is picked up by the camera may bedisplayed corresponding to its position in the real space.

The motion training device may be other than the HMD. For example, itmay be a tablet terminal or a smartphone. In the case where such adisplay terminal is used, for example, the display terminal may bearranged between both hands and motion images which appear as if thetraining target site is moving may be displayed at a different positionfrom the position of the training target site of the user.

The motion training may be carried out for the treatment of phantom limbpain. For example, a patient who has a thumb missing and perceivesphantom limb pain may carry out the motion training for the treatment ofthe phantom limb pain.

In the case where the see-through HMD is used as in the embodiment, thefunction of adjusting the display position of the motion images may beabsent. Even in this case, motion images which appears as if thetraining target site is moving can be displayed at a different positionfrom the position of the training target site of the user if the useradjusts his/her posture.

The motion images IMG may be displayed in the form of an image perceivedas two-dimensional.

In the embodiment, a part or all of the functions and processingrealized by software may be realized by hardware. Also, a part or all ofthe functions and processing realized by the hardware may be realized bysoftware. As the hardware, for example, various circuits such as anintegrated circuit, a discrete circuit, or a circuit module formed by acombination of these circuits, may be used.

The entire disclosure of Japanese Patent Application No. 2016-192596filed Sep. 30, 2016 is expressly incorporated by reference herein.

What is claimed is:
 1. A motion training device comprising: a cameraconfigured to capture an image of a real space containing two body partsof a user; a display configured to display a training target site andmotion images, the training target site corresponding to the two bodyparts of the user, the motion images being a virtual representation ofthe two body parts; and a controller communicably coupled to the cameraand the display and configured to cause the motion images to selectivelyrepeatedly perform desired training movements, wherein the controller isconfigured to; detect a position of the two body parts of the user byextracting the two body parts from the image captured by the camera;determine whether the motion images can be displayed between the twobody parts; display the motion images between the two body parts when itis determined that the motion images can be displayed between the twobody parts; and prompt the user to adjust at least one of the camera andthe two body parts when it is determined that the motion images cannotbe displayed between the two body parts.
 2. The motion training deviceaccording to claim 1, wherein the display is a video see-throughdisplay, and when the controller causes the video see-through display todisplay the motion images between the two body parts, at least part ofthe two body parts in the real space is visible by the user through thevideo see-through display.
 3. The motion training device according toclaim 1, wherein the two body parts correspond to two hands of the user.4. The motion training device according to claim 1, wherein thecontroller causes the motion images to appear smaller than the two bodyparts of the user on the display.
 5. The motion training deviceaccording to claim 1, wherein the controller causes the motion images tobe entirely spaced apart from the two body parts on the display.
 6. Adisplay method comprising: capturing an image of a real space containingtwo body parts of a user by a camera; displaying a training target siteand motion images on a display, the training target site correspondingto the two body parts of the user, the motion images being a virtualrepresentation of the two body parts: causing the motion images toselectively repeatedly perform desired training movements on thedisplay; detecting a position of the two body parts of the user byextracting the two body parts from the image captured by the camera:determining whether the motion images can be displayed between the twobody parts; displaying the motion images between the two body parts whenit is determined that the motion images can be displayed between the twobody parts, and prompting the user to adjust at least one of the cameraand the two body parts when it is determined that the motion imagescannot be displayed between the two body parts.
 7. The display methodaccording to claim 6, further comprising: displaying the motion imagesbetween the two body parts on a video see-through display as thedisplay, wherein at least part of the two body parts in the real spaceis visible by the user through the video see-through display.
 8. Thedisplay method according to claim 6, wherein the two body partscorrespond to two hands of the user.
 9. The display method according toclaim 6, further comprising causing the motion images to appear smallerthan the two body parts of the user on the display.
 10. The displaymethod according to claim 6, further comprising causing the motionimages to be entirely spaced apart from the two body parts on thedisplay.
 11. A display method comprising: providing a camera to a user;providing a display to the user; capturing an image of a real spacecontaining two body parts of the user with the camera; detecting aposition of the two body parts of the user by extracting the two bodyparts from the image captured by the camera; determining whether motionimages can be displayed between the two body parts; displaying themotion images between the two body parts when it is determined that themotion images can be displayed between the two body parts; and promptingthe user to adjust at least one of the camera and the two body partswhen it is determined that the motion images cannot be displayed betweenthe two body parts, wherein the motion images is a virtualrepresentation of at least one of the two body parts.
 12. The displaymethod according to claim 11 wherein the determining step furthercomprises: confirming that the motion images can be displayed entirelybetween the two body parts such that no overlapping exists among themotion images and the two body parts.
 13. The display method accordingto claim 11 further comprising: selectively causing the motion images torepeatedly perform desired training movements.
 14. The display methodaccording to claim 11 further comprising: making the motion imagesappear smaller than the two body parts on the display.